Enhanced perspective view generation in a front curb viewing system

ABSTRACT

A system and method for creating an enhanced perspective view of an area in front of a vehicle, using images from left-front and right-front cameras. The enhanced perspective view removes the distortion and exaggerated perspective effects which are inherent in wide-angle lens images. The enhanced perspective view uses a camera model including a virtual image surface and other processing techniques which provide corrections for two types of problems which are typically present in de-warped perspective images—including a stretching effect at the peripheral area of a wide-angle image de-warped by rectilinear projection, and double image of objects in an area where left-front and right-front camera images overlap.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of the priority date of U.S.Provisional Patent Application Ser. No. 61/826,902, titled ENHANCEDPERSPECTIVE VIEW GENERATION IN A FRONT CURB VIEWING SYSTEM, filed May23, 2013.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates generally to the display of enhanced perspectiveview images of curbs and other structures in front of a vehicle and,more particularly, to a system and method for synthesizing an enhancedperspective view of an area in front of a vehicle using images fromleft-front and right-front cameras, where the enhanced perspective viewis created using a new camera imaging surface model and other processingtechniques which correct distortions, resulting in an image whereartificial magnification of the front-center of the image is corrected,vertical items appear vertical and double-imaging of objects iseliminated in an area where left-front and right-front camera imagesoverlap.

2. Description of the Related Art

Vehicles equipped with forward-view cameras have become increasinglycommon in recent years. Images from forward-view cameras are used for avariety of applications, including lane departure warning and collisionwarning. In these applications, the camera images are not displayed tothe vehicle driver, but rather, the images are analyzed by a vehiclecomputer using image processing techniques, lane boundaries andpotential obstacles are detected from the images, and warnings areissued to the driver as needed.

Even as the advent of new technology has made compact, reliable andaffordable digital cameras a reality, an age-old problem continues tobeset drivers. That problem is the inability of drivers to accuratelyjudge the position of the front of the vehicle relative to an obstacle,such as a curb, when parking. Because of the inherent inability of adriver to see a curb when pulling into a parking space due to theobstruction from the front of the vehicle, the driver is forced toestimate how far to pull forward—hopefully without hitting the curb.This judgment can be difficult, as it relies on past experience,peripheral visual cues and other indirect information. If the driverjudges the forward clearance incorrectly, the vehicle may not be pulledfar enough into the parking space, or worse, the front of the vehiclemay hit the curb, causing damage to the vehicle.

During a parking maneuver, the driver is usually concentrating on manyfacets of parking the vehicle, and may not think to manually turn on afront curb viewing display. However, if a front curb viewing systemautomatically displayed the most appropriate frontal images based on thecontext of the parking maneuver, most drivers would find this visualinformation helpful—particularly if the system not only selected the oneor more views which provide the most assistance to the driver, but alsoproduced enhanced views with corrections for visual distortions whichare inherently present in wide-angle lens images.

There is a need for a forward-view parking assist system which takesadvantage of the available images of curbs and other structures in frontof vehicles, and provides drivers with optimized images which enable thedriver to precisely position the front of the vehicle relative to thecurb or other frontal object.

SUMMARY OF THE INVENTION

In accordance with the teachings of the present invention, a system andmethod are disclosed for creating an enhanced perspective view of anarea in front of a vehicle, using images from left-front and right-frontcameras. The enhanced perspective view removes the distortion andexaggerated perspective effects which are inherent in wide-angle lensimages. The enhanced perspective view uses a camera model including avirtual image surface and other processing techniques which providecorrections for two types of problems which are typically present inde-warped perspective images—including a stretching effect at theperipheral area of a wide-angle image de-warped by rectilinearprojection, and double image of objects in an area where left-front andright-front camera images overlap.

Additional features of the present invention will become apparent fromthe following description and appended claims, taken in conjunction withthe accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a top view of a vehicle including frontal cameras which can beused for curb viewing;

FIG. 2 is a side view illustration of the vehicle of FIG. 1, showingfurther details of the camera arrangement;

FIG. 3 is a schematic diagram of a system for automated and enhancedcurb viewing;

FIG. 4 is a block diagram of a system which is a first embodiment of anenhanced curb viewing system;

FIG. 5 is a block diagram of a system which is a second embodiment of anenhanced curb viewing system;

FIG. 6 is a block diagram of a system which is a third embodiment of anenhanced curb viewing system;

FIG. 7 is a block diagram of the system of FIG. 6, showing how thelearning module operates;

FIG. 8 is a flowchart diagram of a method for providing enhanced frontcurb viewing in a vehicle;

FIG. 9 is an illustration of the display device shown in FIG. 3, showingan example display mode, or arrangement of views on the screen of thedisplay device;

FIG. 10 is an illustration of how vehicle body parts in thebumper/fascia area can create an artificial protrusion effect in avirtual top-down view;

FIG. 11 is an illustration of a virtual top-down view in which theprotrusion shown in FIG. 10 artificially extends forward as a result ofcamera/vehicle geometry and image de-warping, creating an occlusionregion;

FIG. 12 is an illustration of a virtual top-down view in which theocclusion region has been re-textured through image manipulation;

FIG. 13 is a flowchart diagram of a method for eliminating artificialvehicle body part appearance in a virtual top-down view constructed fromvehicle frontal camera images;

FIG. 14 is an illustration of a virtual top-down view in which thecertain portions of the view exhibit low resolution and/or image noise;

FIG. 15 is a flowchart diagram of a method for eliminating lowresolution and image noise in edge regions of a virtual top-down viewconstructed from vehicle frontal camera images;

FIGS. 16A and 16B are illustrations of virtual top-down views from theleft front camera and the right front camera, respectively;

FIG. 17 is an illustration of how some parts of the virtual top-downview can appear to have “double vision” when left and right cameraimages are merged;

FIG. 18 is a flowchart diagram of a first method for eliminating thedouble vision effect for above-ground obstacles in a virtual top-downview constructed from vehicle frontal camera images;

FIG. 19 is a flowchart diagram of a second method for eliminating thedouble vision effect for above-ground obstacles in a virtual top-downview constructed from vehicle frontal camera images;

FIG. 20 is a flowchart diagram of a method for creating an enhancedvirtual top-down view constructed from vehicle frontal camera images;

FIG. 21 is an illustration of a perspective view in front of a vehiclewhich includes distortions inherent in images from wide-angle lenseswhich have been processed using traditional de-warping techniques;

FIG. 22 is a diagram of camera image surface models which may be used tocorrect the distortions seen in FIG. 21;

FIG. 23 is an illustration of a perspective view in front of a vehiclein which the distortions seen in FIG. 21 have been corrected byapplication of the new virtual image surface model of FIG. 22 and otherprocessing techniques; and

FIG. 24 is a flowchart diagram of a method for creating an enhancedperspective view constructed from vehicle frontal camera images usingthe camera image surface model of FIG. 22.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The following discussion of the embodiments of the invention directed toa system and method for providing enhanced perspective views of an areain front of a vehicle is merely exemplary in nature, and is in no wayintended to limit the invention or its applications or uses. Forexample, the invention described below has particular application forfront curb view imaging for vehicles. However, the invention is alsoapplicable to other motorized devices, such as forklifts, and may alsobe useful in rear-view or side-view situations.

FIG. 1 is a top view of a vehicle 100 including frontal cameras whichcan be used for curb viewing. The vehicle 100 includes a pair of cameras102, 104 positioned behind the grill or in the front bumper of thevehicle 100. The first (or left) camera 102 is spaced apart by ahorizontal or lateral distance 106 from the second (or right) camera104. The distance 106 will vary depending upon the make and model of thevehicle 100, but in some embodiments may be approximately one meter. Thecameras 102 and 104 have an optical axis 108 that is offset from theforward direction of the vehicle by a pan angle θ. The angle θ which isemployed may vary depending upon the make and model of the vehicle 100,but in some embodiments is approximately 10°. Whatever orientation isselected for the cameras 102 and 104, each camera captures an ultra-widefield of view (FOV) using a fish-eye lens to provide approximately a180° FOV that partially overlaps at area 110. The FOV of the camera 102is represented by sector 112, and the FOV of the camera 104 isrepresented by sector 114. The images captured by the cameras 102 and104 may be received by a video acquisition device 116, and the imagesfurther processed in a processor 118 having image processing hardwareand/or software as will be discussed in detail below, to provide one ormore types of driver assisting images on a display device 120.

The processor 118 is a computing device including at least amicroprocessor and a memory module, of any type commonly used invehicles. The processor 118 may be a general purpose device which alsoperforms other computing functions, or the processor 118 may be a customdesign which is configured specifically for front curb view imageprocessing. In any case, the processor 118 is configured to perform thesteps of the methods discussed herein. That is, the methods for enhancedcurb viewing are intended to be carried out on a computing device suchas the processor 118, not in a person's head or using paper and pencil.

FIG. 2 illustrates a side view of the vehicle 100. The right camera 104is shown positioned by a distance 130 above the road/ground. Thedistance 130 will depend upon the make and model of the vehicle 100, butin some embodiments is approximately one-half meter. Knowing thedistance 130 is useful for computing virtual images from the field ofview 114 to assist the driver of the vehicle 100. The camera 104 (andcamera 102 on the opposite side of the vehicle) is typically alignedsuch that it is slightly angled downward by a tilt angle φ to providethe field of view 114 as shown. The angle φ will vary by make and modelof the vehicle 100, but in some embodiments may be approximately 10°.

As discussed above, the dual front camera arrangement of FIGS. 1 and 2is common on vehicles today, with the images typically being used incollision avoidance and lane departure warning applications. As alsodiscussed above, the front camera images can also be used for curbviewing as a driver aid. However, until now, there has not been anautomated system for providing enhanced front curb viewing in situationswhere desired by the driver.

FIG. 3 is a schematic diagram of a system 140 for automated and enhancedcurb viewing. The system 140 includes elements discussed above withrespect to FIG. 1, including the front cameras 102 and 104, the videoacquisition device 116, the processor 118 and the display device 120.FIG. 3 provides a high-level depiction of the system 140, showing thebasic elements. As could be expected, the processor 118 can include manydifferent functions and modules, which will be discussed in the figuresthat follow.

FIG. 4 is a block diagram of a system 160 which is a first embodiment ofan enhanced curb viewing system. This first embodiment is a simpledesign in which a user command is the primary trigger for initiatingfront curb viewing. A user input 162 provides a signal from the driverof the vehicle 100 if the driver wants to view front curb images. Theuser input 162 may be a physical button, a switch, a virtual button on atouch screen device such as the display device 120, an audio microphonefor detecting a spoken command, or other type of input. When the driveractivates the user input 162, this is taken as a trigger to providefront curb images on the display device 120, but only if the vehiclespeed is below a certain threshold, which may be a regulatoryrequirement. A vehicle speed signal is provided from a vehicle statemodule 164, which may include data from various vehicle sensors,including vehicle speed, steering angle, yaw rate, longitudinal andlateral acceleration, etc.

If the user input 162 has been activated and the vehicle speed is belowthe threshold, then a trigger module 166 sets a front curb viewing imagestatus to yes. A switch 168 determines what to display on the displaydevice 120. If the front curb viewing image status is equal to yes, thenthe switch 168 commands a processing module 170 to compute enhancedfront curb viewing images using image input from the video acquisitiondevice 116. Finally, a mode selection module 172 determines what view orviews should be displayed on the display device 120. In the simpleembodiment of the system 160, the mode selection module 172 may select adefault arrangement of images (including, for example, a forwardperspective view and a virtual top-down curb view—discussed in detaillater). The mode selection module 172 may also support user selection ofviewing mode—by way of the user input 162, for example.

If the front curb viewing image status is equal to no, then the switch168 passes control to a display control module 174, which displays otherinformation on the display device 120. For example, when the driver isnot viewing front curb images, the display device 120 could serve as aradio or other audio/video interface, or a navigation map display, orany other appropriate function.

The trigger module 166, the switch 168, the processing module 170 andthe mode selection module 172 can all be considered to be included inthe processor 118 discussed previously. That is, the elements 166-172,which each perform a separate function, do not need to be separatephysical devices, but rather can be embodied in hardware or softwarewhich is all included in the processor 118.

FIG. 5 is a block diagram of a system 180 which is a second embodimentof an enhanced curb viewing system. This second embodiment includesadditional triggering signal sources for initiating front curb viewing.The user input 162 still offers a direct means for the driver of thevehicle 100 to indicate that he/she wants to view front curb images, ifthe vehicle speed is below the threshold. However, in addition to theuser input 162, a heuristics module 182 uses various data sources toevaluate the vehicle's operating environment, and applies rules todetermine when front curb viewing is likely to be desired by the driver.The heuristics module 182 includes the vehicle state module 164discussed previously, which not only provides a vehicle speed signal forcomparison to the speed threshold, but may also provide steering angle,yaw rate, longitudinal and lateral acceleration and other data. Rulescan be defined which trigger front curb viewing when, for example,vehicle speed is low and steering angle is high, indicating a likelyparking lot maneuver.

The heuristics module 182 also includes an object detection module 184,which receives camera images from the video acquisition device 116. Theobject detection module 184 analyzes the front camera images to detect aparking spot, a curb, or any other object ahead of the vehicle 100. If aparking spot is detected, and if other vehicle parameters such as speedare appropriate, it may be inferred that the driver is maneuvering thevehicle 100 to park in the detected parking spot. This would cause theheuristics module 182 to issue a trigger signal to activate front curbviewing. Even in absence of a detected parking spot, the presence of acurb ahead of the vehicle 100 may also cause the heuristics module 182to issue a trigger signal to activate front curb viewing. The curb couldbe a stand-alone unit with a definite left and right end, or it could bea continuous raised sidewalk as is common in many commercial buildingparking lots. Likewise, any other object detected immediately ahead ofthe vehicle 100—such as a wall, one or more posts, a fence, etc.—maytrigger front curb viewing signal from the object detection module 184of the heuristics module 182.

The heuristics module 182 also includes a GPS/map module 186, which usesvehicle position data from an onboard Global Positioning System (GPS)receiver, along with data from a digital map and/or navigation system,to determine when the vehicle 100 is likely to be in a parkingsituation. For example, it can easily be determined from GPS and digitalmap data that the vehicle 100 is in a parking lot at a mall, or in aparking lot at an office building. This knowledge may be used to begin asearch for a parking spot or curb using the object detection module 184,or the knowledge may be used to directly trigger front curb viewing ifvehicle parameters such as speed and steering angle (from the vehiclestate module 164) meet certain criteria.

The heuristics module 182 evaluates all available data, as discussedabove, and determines whether front curb viewing is likely to be desiredby the driver, based on a set of pre-established rules. Of course, ifcurb viewing is selected via the user input 162, then front curb viewingis activated (as long as the vehicle speed is below the threshold),regardless of what determination is made by the heuristics module 182.Likewise, if the driver indicates via the user input 162 that curbviewing is not wanted, then front curb viewing is deactivated,regardless of what determination is made by the heuristics module 182.

The user input 162 and the heuristics module 182 provide signals to thetrigger module 166, which sets the front curb viewing image status toyes or no depending on the signals. As in the first embodiment, theswitch 168 determines what to display on the display device 120. If thefront curb viewing image status is equal to yes, then the switch 168commands the processing module 170 to compute enhanced front curbviewing images using image input from the video acquisition device 116.Finally, the mode selection module 172 determines what view or viewsshould be displayed on the display device 120.

In the second embodiment represented by the system 180, the modeselection module 172 can select a preferred display mode based on thecontext of the situation as defined by data from the heuristics module182. For example, if the vehicle 100 is being steered to the right, orif a parking spot is detected ahead and to the right of the vehicle 100,then a front-right perspective view may be chosen for the display mode,possibly along with a virtual top-down view. Many combinations of viewsmay be defined in the display modes. Likewise, many different factorsmay be considered in the logic for selecting the display mode, with theobjective being to display the view or views which provide the mostdriver assistance in the context of the current parking situation. Themode selection module 172 may also support user selection of viewingmode—by way of the user input 162, for example.

If the front curb viewing image status is equal to no, then the switch168 passes control to the display control module 174, which displaysother information on the display device 120, as discussed previously.The trigger module 166, the switch 168, the processing module 170 andthe mode selection module 172 can all be considered to be included inthe processor 118, as also discussed previously. In addition, theheuristics module 182 of the system 180 can also be included in theprocessor 118.

FIG. 6 is a block diagram of a system 200 which is a third embodiment ofan enhanced curb viewing system. The system 200 includes all of theelements of the system 180 (the second embodiment) discussed above.However, this third embodiment includes yet another triggering signalsource for initiating front curb viewing. A behavior/history learningmodule 202 uses machine learning techniques to determine when it islikely that the driver of the vehicle 100 wants to view front curbimages. The learning module 202 provides a signal to the trigger module166, which sets the front curb viewing image status to yes or nodepending on three signal sources. The learning module 202 may alsolearn driver preferences for display modes—that is, what view or viewsto display in what parking situations—and may provide that data to themode selection module 172 (via the trigger module 166, as shown in FIG.6, or directly).

FIG. 7 is a block diagram of the system 200 showing how the learningmodule 202 works. Elements of the system 200 which do not interact withthe learning module 202—specifically, everything below the triggermodule 166—are omitted from FIG. 7 for simplicity. The learning module202 includes both off-line or preprogrammed learning and adaptivelearning. A database 204 contains data representing a general populationof drivers. The database 204 is preprogrammed and is available when thevehicle 100 is new. The database 204 is not continuously updated, butmay be periodically updated (via a telematics service, for example) withnew data for a general population of drivers. Data from the database 204is used in an off-line learning module 206 to make a first determinationof whether front curb viewing is desired, and this is communicated via asignal from the off-line learning module 206 to the learning module 202.

A database 208 continuously collects data from the user input 162, theheuristics module 182 and the trigger module 166 during vehicleoperation. The data from the database 208 is used in an adaptivelearning module 210 to make a second determination of whether front curbviewing is desired, and this is communicated via a signal from theadaptive learning module 210 to the learning module 202. The adaptivelearning module 210 applies machine learning techniques to the data fromthe database 208 to determine whether front curb viewing should betriggered in a current situation based on whether front curb viewing wastriggered in similar circumstances previously. For example, it may beapparent in the data that the driver of the vehicle 100 frequentlyactivates front curb viewing at a certain vehicle location, even thoughmap data does not identify the location as a parking lot. In thissituation, when the vehicle 100 again arrives at the location, and whenvehicle speed is below the threshold, the adaptive learning module 210can trigger front curb viewing. This analysis can be further enhanced toinclude recognition of a specific pattern of steering maneuvers whichthe driver may typically follow to reach a certain parking space.

As another example, the adaptive learning module 210 may determine thatthe driver of the vehicle 100 typically does not want front curb viewingwhen approaching curbs and sidewalks (maybe because the vehicle 100 hashigh ground clearance), but the driver does want front viewing whenapproaching posts, walls and fences. This pattern can be detected in thedata from the object detection module 184 and the user input 162. All ofthese types of determinations can be made by evaluating not just thedata from the three parts of the heuristics module 182, but also datafrom the user input 162 and the trigger module 166. That is, it isimportant to consider not only the circumstances in which front curbviewing was previously activated, but also to consider the driver'spreferences as indicated by manually activating or deactivating frontcurb viewing via the user input 162.

The adaptive learning module 210 can also distinguish between differentdrivers of the vehicle 100, by way of a key fob identifier or otherdriver recognition techniques, and can adapt accordingly. For example,two drivers of the vehicle 100 may have very different preferences forfront curb viewing, where one driver may never want to use it and theother driver may use it frequently. On the other hand, both drivers ofthe vehicle 100 may have the same “home” location, and it may bedetermined that both drivers prefer to activate curb viewing at the homelocation. These types of driver-specific and/or vehicle-specificdeterminations can easily be made by the adaptive learning module 210.The ability to learn the preferences of drivers of the vehicle 100 as itis driven distinguishes the adaptive learning module 210 from theoff-line learning module 206 and the heuristics module 182.

FIG. 8 is a flowchart diagram 220 of a method for providing enhancedfront curb viewing in a vehicle. At box 222, a vehicle speed signal isread from a vehicle speed sensor. At decision diamond 224, it isdetermined whether the vehicle speed is below a speed threshold, whichmay be defined by government regulation for displaying front curb viewimages. If the vehicle speed is not below the threshold, the methodloops back to the box 222 until such time as the vehicle speed is belowthe threshold. If the vehicle speed is below the threshold at thedecision diamond 224, then at decision diamond 226 it is determinedwhether the vehicle driver has selected front curb viewing via the userinput 162.

If the driver has not selected front curb viewing, then at box 228, thevehicle operating environment is evaluated to determine whether anyfactors indicate that front curb viewing should be activated. Thevehicle operating environment evaluation is performed by the heuristicsmodule 182, and includes factors such as vehicle speed and steer angle,the presence of a parking spot, a curb or other obstacle in front of thevehicle, and vehicle location as indicated by GPS and digital map. Atbox 230, the driving behavior and history of the vehicle is evaluated bythe learning module 202 to determine whether front curb viewing shouldbe triggered in a current situation based on whether front curb viewingwas triggered in similar circumstances previously. At decision diamond232, the signals from the boxes 228 and 230 are evaluated to determinewhether any triggers exist which indicate that front curb viewing shouldbe activated. If no such triggers exist, then at terminus 234 theprocess ends and other information (such as radio or navigation) isdisplayed on the display device 120.

If the driver has selected front curb viewing at the decision diamond226, or triggers exist at the decision diamond 232 which indicate thatfront curb viewing should be activated, then at box 236 the videosignals from the video acquisition device 116 are processed to produceenhanced front curb viewing images. At box 238, a display mode for frontcurb viewing is selected, based on past driver preference and/or thecontext of the current parking situation, as discussed previously. Atbox 240, the enhanced front curb view images are displayed according tothe selected display mode.

Using the techniques described above, an enhanced front curb viewingsystem can be delivered which provides the vehicle driver with valuablevisual assistance for maneuvering near curbs and other obstacles infront of a vehicle. The enhanced front curb viewing system uses anexisting display device and cameras, and provides the driver with visualinformation which would otherwise be unavailable—thereby resulting infar fewer incidents of vehicle damage due to contact with frontalobstacles.

As can be understood from the above discussion, the selection anddisplay of certain frontal views is essential to the effectiveness of afront curb viewing system. Views can include both enhanced versions ofnative camera views (such as a forward perspective view from the left orright camera) and “virtual views”, which are views which are synthesizedby processing multiple camera images to produce a view perspective whichnone of the cameras records directly.

FIG. 9 is an illustration of the display device 120, showing an exampledisplay mode, or arrangement of views on the screen of the displaydevice 120. The example shown in FIG. 9 includes at least three imageviews, where the right side of the display device 120 includes a view250, which is a forward perspective view (enhanced perspective viewcreation is discussed below). The left side of the display device 120includes a bird's eye view of the area around the vehicle 100. View 252is a virtual top-down front view (which is a view synthesized from theimages from the cameras 102 and 104), and view 254 is a virtual top-downrear view (which may be useful if the vehicle is maneuvering forward andbackward during parking). A depiction of the vehicle 100 is providedbetween the view 252 and the view 254 for reference. Side views 256 canbe constructed from images from side view cameras if available, or canbe constructed using temporal filling techniques (discussed below) fromfront and rear camera images. As can easily be envisioned, manydifferent combinations of views can be defined as display modes, whichare selected as discussed in detail above.

When the front bumper of the vehicle 100 is within two feet or so of acurb, many drivers may find a top-down front view to be most helpful. Infact, during forward parking maneuvers, many drivers may prefer adisplay mode where an enhanced top-down front view occupies the fullscreen of the display device 120. Techniques for processing digitalimages are known in the art—including constructing virtual image views,and de-warping images from wide-angle or fish-eye lenses to remove thedistortion. However, creation of a virtual top-down view from wide-anglelenses which are aimed more nearly horizontal than vertical creates aset of challenges which until now have not been solved.

Three types of problems have been identified which are associated withvirtual top-down views synthesized from images from the cameras 102 and104. These three classes of problems, the solutions of which arediscussed in detail below, are as follows;

1) Body part artifacts in de-warped image2) Low resolution and/or noise around boundaries of de-warped image3) “Double vision” effect in surfaces and objects above ground level

FIG. 10 is an illustration of how vehicle body parts in thebumper/fascia area can create an artificial protrusion effect in avirtual top-down view. In FIG. 10, it can be seen that the camera 104 ismounted so that the lens is very nearly flush with the face of thebumper of the vehicle 100. However, many vehicles include protrusionsfrom the lower front bumper or fascia. These protrusions, such as theprotrusion 260, extend forward of the lens of the camera 104 andobstruct the view of the ground from the camera 104. Thus, in a virtualtop-down view, the protrusion 260 will appear to be farther forward andlarger than it would be in a true top-down view. The difference betweenthe apparent size/position of the protrusion 260 and the truesize/position in a top-down view is represented by a distance 262. Thedistance 262 is a function of the geometry of the camera, the protrusion260, and the distance 130 above ground, and can be compensated for.

FIG. 11 is an illustration of a virtual top-down view in which theprotrusion 260 artificially extends forward as a result ofcamera/vehicle geometry and image de-warping, creating an occlusionregion 280. The virtual top-down view, shown as being displayed on thedisplay device 120, depicts a horizontal parking lot surface 270, ahorizontal walkway surface 272 which is raised some height above thesurface 270, and a vertical face 274 which acts as a curb. Thisconfiguration of a parking lot surface and a raised curb/walkway iscommon in many commercial and retail parking lots. Joints 276 representjoints in the concrete sections of the walkway surface 272, where thejoints 276 become vertical on the vertical face 274, and the joints 276may extend for some distance into the parking lot surface 270.

In the virtual top-down view on the display device 120, the bottom edgerepresents the true front of the vehicle 100. Thus, as the vehicle 100is slowly driven forward, the driver can see when an obstacle isreached. As discussed above, the appearance that the protrusion 260extends significantly forward is artificial, with the occlusion region280 being an artifact of image de-warping and the natural geometricrelationships of the camera lens and the protrusion 260 relative to eachother and to ground. Techniques have been developed which allow theocclusion region 280 to be modified so that the protrusion 260 appearsin its true position in a synthesized virtual top-down view, and theocclusion region 280 is synthetically textured to appear as thesurrounding ground. FIG. 12 is an illustration of a virtual top-downview in which the size of the occlusion region 280 has beensignificantly reduced to reflect the true position of the protrusion260, and most of the area which was originally included in the occlusionregion 280 has been re-textured through image manipulation.

FIG. 13 is a flowchart diagram 300 of a method for eliminatingartificial vehicle body part appearance in a virtual top-down viewconstructed from vehicle frontal camera images. Creation of the enhancedvirtual top-down view shown in FIG. 12 begins at box 302 by quantifyingthe geometric relationships discussed above and shown in FIG. 10. Thesethree-dimensional relationships are known, unchanging vehicle designquantities for any particular vehicle model. It therefore becomes afairly simple calculation to determine exactly which pixels of a cameraimage will be occluded by the protrusion 260. It is also possible tocompare a true top-down view of the front of the vehicle 100 with thevirtual top-down view shown in FIG. 11, and manually identify the pixelswhich are artificially occluded by the protrusion 260.

At box 304, a de-warping calculation is performed, to account for thedistortion of the image when the naturally warped image from thewide-angle lens of the cameras 102 and 104 is flattened or “de-warped”to make straight lines appear straight. Image de-warping calculationsare known in the art. At box 306, a set of pixels which are artificiallymasked by the vehicle body protrusion 260 are provided. The pixels fromthe box 306, which are the occlusion region 280 of FIG. 11, will need tobe re-textured to appear as ground surface.

At box 308, the same pixels in the occlusion region 280 are analyzed forre-texturing. Two techniques can be used for the filling andre-texturing of the pixels in the occlusion region 280. At box 310, aspatial filling is performed. The spatial filling at the box 310includes identifying texture (such as the grainy gray appearance ofweathered concrete) and structure (such as the joint 276) near theocclusion region 280, and filling in the occlusion region 280 based onthe surrounding texture and structure. The spatial filling techniquefrom the box 310 can be very effective when the occlusion region 280 isfairly small, and/or the texture of the parking lot surface 270 isfairly consistent.

At box 312, a temporal filling is performed. The temporal filling at thebox 312 involves using actual image pixels of the parking lot surface270, obtained when those pixels were just outside the occlusion region280, and transposing the pixels into the occlusion region 280 based onvehicle motion. Vehicle motion for the temporal filling can bedetermined from optical flow or from vehicle dynamics sensors, whereoptical flow involves following the motion of feature points assubsequent images are obtained. Using vehicle motion, which is typicallyslow and steady during a parking maneuver, pixels representing theactual part of the parking lot surface 270 which is within the occlusionregion 280 can be copied from previous images into subsequent images ofthe virtual top-down view.

At box 314, the occlusion region 280 is filled and re-textured using acombination of the spatial filling technique from the box 310 and thetemporal filling technique from the box 312. At box 316, an enhancedvirtual top-down view is created in which the body mask region from thebox 306 is in-filled with surrounding background texture from the box314, so that only the true shape of the protrusion 260 remains in theview. As can be seen in FIG. 12, the appearance of the protrusion 260 isreduced by using the filled and re-textured occlusion region 280.

The second type of problem associated with virtual top-down viewssynthesized from left- and right-front camera images is low resolutionand/or noise around boundaries of de-warped image. FIG. 14 is anillustration of a virtual top-down view in which the certain portions ofthe view can exhibit low resolution and/or image noise. The virtualtop-down view of FIG. 14, like FIGS. 11 and 12, depicts the horizontalparking lot surface 270, the horizontal walkway surface 272 which israised some height above the surface 270, the vertical face 274 whichacts as a curb, and the joints 276 in the concrete sections of thewalkway surface 272.

As discussed above, it is necessary to “de-warp” images from wide-anglelenses in order to remove distortion and create a realistic lookingtop-down view. When de-warping images, resolution is inherently lost inperipheral or edge regions of the virtual view, as each pixel in theseregions is magnified to occupy more of the image. Left and rightperipheral regions 320 and upper peripheral region 322 as shown in FIG.14 represent the low-resolution regions of the virtual top-down view.Image/video noise in the image is also a problem in the regions 320 and322, as a small light or dark speck could be magnified to appear muchlarger than it really is. Of course, the boundaries of thelow-resolution regions are not perfectly straight as shown in FIG.14—but the regions 320 and 322 are illustrative of the concept. There isno low-resolution region in the lower part of the image, as this edge isnearest the cameras 102 and 104, and the pixels in this part of theimage do not need to be significantly magnified in the de-warpingprocess.

FIG. 15 is a flowchart diagram 340 of a method for eliminating lowresolution and image noise in edge regions of a virtual top-down viewconstructed from vehicle frontal camera images. At box 342, a de-warpingcalculation is performed on the virtual top-down view, to account forthe distortion of the image when the naturally warped image from thewide-angle lens of the cameras 102 and 104 is flattened to make straightlines appear straight. At box 344, a set of pixels which areartificially low in resolution are provided. These pixels represent theedge regions 320 and 322, where the low resolution and noise are theresult of pixel magnification in the de-warping process.

Two techniques can be used for the filling and re-texturing of thepixels in the low resolution regions 320 and 322. At box 346, a spatialresolution enhancement and de-noise calculation is performed. Thespatial enhancement at the box 346 includes identifying both color andintensity channels in the low-resolution regions 320 and 322, and usingthe native colors and intensities in a smoothing calculation whichpreserves edges such as the edges of the vertical face 274. The spatialfilling at the box 346 also includes borrowing texture from neighboringfull-resolution areas, such as region 324 which can be used for theleft-side low-resolution region 320 and the left portion of the upperlow-resolution region 322. Using the surface texture from theneighboring full-resolution region 324, and color and intensity patternsfrom the low-resolution regions 320 and 322 themselves, the spatialresolution enhancement and de-noise calculation at the box 346 cansignificantly restore image quality around the edges of the virtualtop-down view.

At box 348, a temporal resolution enhancement and de-noise calculationis performed. The temporal enhancement at the box 348 involves usingactual image pixels of the surface 270, 272 and 274, obtained when thosepixels were just outside the low-resolution regions 320 and 322, andtransposing the pixels into the low-resolution regions 320 and 322 basedon vehicle motion. As discussed previously, vehicle motion for thetemporal enhancement can be determined from optical flow or from vehicledynamics sensors, where optical flow involves following the motion offeature points as subsequent images are obtained. Vehicle motion tendsto work against temporal enhancement of the region 322 in particular, aspixels in that region will not have previously been in a high-resolutionregion of the image if the vehicle 100 has been moving only forward.However, temporal enhancement may be helpful for the left and rightlow-resolution regions 320, particularly if the vehicle 100 is turningwhile maneuvering to park. Using vehicle motion, pixels representingactual parts of the surfaces 270-274 which are within the low-resolutionregions 320 can be copied from previous images into subsequent images ofthe virtual top-down view at the box 348.

At box 350, the low-resolution regions 320 and 322 are redefined using acombination of results of the spatial enhancement technique from the box346 and the temporal enhancement technique from the box 348. At box 352,an enhanced virtual top-down view is created in which the originalpixels from the low-resolution regions 320 and 322 are replaced with theredefined pixels from the box 350, and combined with the remainder ofthe de-warped image from the box 342.

The third type of problem associated with virtual top-down viewssynthesized from left- and right-front camera images is a “doublevision” effect in objects which protrude above ground level. FIGS. 16Aand 16B are illustrations of virtual top-down views from the left frontcamera 102 and the right front camera 104, respectively, and FIG. 17 isan illustration of how some parts of the virtual top-down view canappear to have “double vision” when left and right camera images arejoined. In the FIGS. 16 and 17, the surfaces 270, 272 and 274 appear asin previous figures, as do the joints 276 (there are two differentjoints 276—one to the left, and one to the right—as shown in FIGS. 11,12 and 14). In FIGS. 16A and 16B, a crack 360 can be seen in the walkwaysurface 272. The crack 360 appears toward the right side of the leftcamera image in FIG. 16A, and the crack 360 appears toward the left sideof the right camera image in FIG. 16B.

In FIG. 17, the crack 360 appears twice, even though there is only onecrack 360 in the walkway surface 272. This “double vision” effect is aresult of the left and right image combination being based oncalculations which assume everything in the image is at the same groundlevel. Objects or obstacles which extend above ground level (which is atthe lower edge of the image, that being the portion of the image nearestthe cameras 102 and 104, and is known to be the distance 130 below thecameras) can appear out of position in the de-warped and combinedvirtual top-down view. However, this double vision effect can becorrected in the enhanced virtual top-down view, via the calculationsdiscussed below.

FIG. 18 is a flowchart diagram 380 of a first method for eliminating thedouble vision effect for above-ground obstacles in a virtual top-downview constructed from vehicle frontal camera images. At box 382, ade-warping calculation is performed on the virtual top-down viewconstructed from left- and right-front camera images, to account for thedistortion when the naturally warped images from the wide-angle lens ofthe cameras 102 and 104 is flattened to make straight lines appearstraight. At box 384, curbs and raised walkways are detected in theconstructed virtual top-down view. As discussed previously, theseparking space boundaries can be detected by analyzing a sequence ofimages during vehicle motion, looking for straight lines and othervisual indicators of curbs and raised walkways.

At box 386, above-ground obstacles are detected in the images from thecameras 102 and 104 using stereo vision and/or structure from motiontechniques. Structure from motion (SFM) refers to the process ofestimating three-dimensional structures from two-dimensional imagesequences which may be coupled with local motion signals. The stereovision and SFM analysis provides information about the size, shape andlocation of above-ground obstacles in the forward path of the vehicle100. At box 388, other sensors onboard the vehicle 100 provide objectdetection signals for the area ahead of the vehicle 100. The otheronboard sensors may include—but are not limited to—ultrasonic,short-range radar and LIDAR, for example.

At junction 390, the frontal obstacles identified in the boxes 384-388are scaled and combined, to allow all of the data to be used to definean object map in the area ahead of the vehicle 100. At box 392, acomplete model of the three dimensional objects—including curbs, raisedwalkway, posts, etc.—in front of the vehicle 100 is assembled from thedata from the boxes 384-388. The model of the three dimensional objectsincludes the location, size and shape of the objects. Where curbs andraised walkways are present, it is extremely valuable to be able todetermine the height of these objects, for at least two reasons. First,if a curb or raised walkway has a height of six inches above the parkinglot surface, and the vehicle 100 only has a front air dam groundclearance of five inches, warnings can be given to the driver toindicate that the vehicle 100 should be stopped before the front bumperreaches the curb or raised walkway (as opposed to driving forward untilthe front tires bump to a stop). Second, knowing the height of theraised walkway 272 above the parking lot surface 270 allows for acalculation to be made which eliminates the double image of the crack360 as seen in FIG. 17.

At box 394, images of curbs or other above-ground obstacles arerendered, using the 3D model of objects from the box 392 and theoriginal de-warped image data from the box 382. By knowing the location,size and shape of any above-ground obstacles ahead of the vehicle 100,the left- and right-front camera images can be reliably combined withoutdisplaying duplicate copies of the obstacles. For example, consider asituation where a post is present in front of the vehicle 100, to theleft of center. If the left- and right-front camera images were simplyde-warped and combined, there would appear to be two such posts, eachwith a different apparent distance and location, in the combined image.However, with the knowledge from the 3D object model that there is onlyone such post, and the knowledge of the actual location of the post, theleft- and right-front camera images can be processed such that the postcorrectly appears only once. This same sort of image processing can beused to correctly render any above-ground feature, including the crack360 of FIG. 17, which will appear only once.

Finally, at box 396, the refined virtual top-down view is synthesized,including the above-ground obstacle renderings from the box 394. Therefined virtual top-down view does not include false double-images ofabove-ground items like the crack 360, and the refined view alsoimproves the visual rendering of 3D objects based on their actual size,shape and location as modeled using all available data sources.

FIG. 19 is a flowchart diagram 400 of a second method for eliminatingthe double vision effect for above-ground objects in a virtual top-downview constructed from vehicle frontal camera images. At box 402, ade-warping calculation is performed on the virtual top-down viewconstructed from left- and right-front camera images, to account for thedistortion when the naturally warped images from the wide-angle lens ofthe cameras 102 and 104 is flattened to make straight lines appearstraight. At box 404, the de-warped image is analyzed to detect anydouble image effect in texture or structure. For example, image analysistechniques are known which could readily recognize that the crack 360,which appears twice in FIG. 17, is really the same object.

If double image effects are found at the box 404, then at decisiondiamond 406 the process loops back to box 408 where an above-groundheight of the double-imaged structure is estimated based on the lateraldistance between the duplicate images. The process then returns to thebox 402 where the de-warp calculation is performed again, using arevised assumption about above-ground height for the portion of thefield of view containing the double-imaged structure. The revisedassumption about above-ground height will modify the de-warpcalculations, causing a lateral shift of objects in that portion of thefield of view.

The image is then re-analyzed at the box 404 and, if necessary, theheight assumption is further revised. This continues until the doubleimage effect is eliminated and, at the decision diamond 406, the processmoves to box 410 where the final refined virtual top-down view isoutput.

Other, simpler methods of eliminating the “double vision” effect inobjects which protrude above ground level, as shown in FIGS. 16 and 17,are also possible. In one such simpler method, rather than iterativelyestimating curb height as in the method of FIG. 19 discussed above, aset of standard curb heights (such as 5, 6, 7 and 8 inches) can beassumed, and the image transformation mapping for each of these heightscan be pre-calculated. Then, when a double-imaging effect is detected ina merged image, the image transformation calculation can quickly bere-run for all of the standard curb heights, and the results can beanalyzed to determine which curb height transformation did the best jobof eliminating the double-imaging effect. This method avoids theiterative calculations discussed previously and directly provides afinal image with minimal double-image effects, while also providing anestimate of curb height which could be used in other vehicle systems.

Yet another method of eliminating the “double vision” effect in objectswhich protrude above ground level is to simply compare the left andright images, find common feature points in the overlap area of theimages, and use image registration to transform the images so that thecommon features points line up in the overlap area when the images aremerged. This method does not provide an estimate of curb height, but maybe the simplest to perform computationally, and still provides a finalimage with minimal double-image effects.

FIG. 20 is a flowchart diagram 420 of a method for creating an enhancedvirtual top-down view constructed from vehicle frontal camera images. Atbox 422, a de-warping calculation is performed on the virtual top-downview constructed from left- and right-front camera images, to accountfor the distortion when the naturally warped images from the wide-anglelens of the cameras 102 and 104 is flattened to make straight linesappear straight. At box 424, artificially exaggerated vehicle body partartifacts are removed from the de-warped image, using the methods of theflowchart diagram 300 shown in FIG. 13. At box 426, low resolution andimage/video noise around edges of the de-warped image are corrected,using the methods of the flowchart diagram 340 shown in FIG. 15. At box428, the “double vision” effect in surfaces or objects above groundlevel is corrected, using one of the methods shown in FIG. 18 or FIG. 19or one of the simpler methods subsequently described. The result of theimage enhancements from the boxes 424-428 is a virtual top-down viewwhich is significantly more realistic in appearance than an ordinaryde-warped image.

Using the techniques described above, three problems which are typicallyassociated with virtual top-down views synthesized from multiple cameraimages can be solved, resulting in a very realistic virtual top-downfront curb view. In prototype system tests, the enhanced virtualtop-down view has been demonstrated to be extremely helpful to driversmaneuvering near curbs and other frontal obstacles. The virtual top-downview synthesis techniques discussed above in the context of a front vieware also applicable to virtual top-down views to the rear and to thesides of the vehicle 100, if camera images are available. The techniquescan also be used with more than two cameras per virtual view. If morethan two cameras are available to construct a virtual top-down frontview, for example, the resultant view quality will improve because moreimage pixels will be used as input and image de-warping effects can bereduced.

As discussed above, a forward perspective view may be selected as thelargest view on the display device 120, as drivers are typicallycomfortable looking at a perspective view. Techniques for processingdigital images are known in the art—including de-warping images fromwide-angle lenses to remove the distortion which creates a fish-eyeeffect where objects in the front center of the image are artificiallymagnified and objects toward the periphery of the image are diminishedand slanted away from center. However, traditional de-warping techniquesand camera models can create other problems in the processed images.

FIG. 21 is an illustration of a perspective view in front of the vehicle100 which includes distortions created by traditional de-warpingtechniques. FIG. 21 is a perspective view of essentially the same sceneshown in previous figures—including the horizontal parking lot surface270, the horizontal walkway surface 272 which is raised some heightabove the surface 270, the vertical face 274 which acts as a curb, andthe joint 276 in the concrete sections of the walkway surface 272. Theperspective view also includes an above-horizon area 440, which includesbackground objects 442 such as trees and telephone poles.

Two main types of problems have been identified which are associatedwith virtual perspective views synthesized from images from the cameras102 and 104 when processed with traditional camera models. These twoclasses of problems, the solutions of which are discussed in detailbelow, are as follows;

1) Stretching and low resolution effect at peripheral part of de-warpedwide-angle image, because of the rectilinear projection of pinholemodel.2) “Double vision” effect in overlapped field of view region and missingobjects in blind spot of merged images.

The first problem with perspective views which have been processed usingtraditional de-warping by rectilinear projection using a pinhole cameramodel—stretching and low resolution—is apparent in two places in FIG.21. First, the image resolution is decreased in area 444 due tostretching. Second, vertical joint 446, which is the vertical portion ofthe joint 276, appears to be sloped rather than vertical. In addition,there can be an exaggerated appearance of body part protrusions, such asthe protrusion 260 discussed previously, due to this stretching. Theseproblems can all be addressed by using new camera image surface models.

FIG. 22 is a diagram of camera image surface models which may be used tocorrect the distortions seen in FIG. 21. A traditional pinhole cameramodel 460 is shown, in which light from a scene passes through a pinholeaperture and impinges on an imaging surface. In the traditional model460, the imaging surface is a flat plane. This type of model results inthe stretching effects in the peripheral area of the de-warped imageshown in FIG. 21. In order to correct this stretching effect, an imagingsurface 462 can be defined which is in the shape of a horizontalhalf-cylinder. Applying a model of the imaging surface 462 to thevirtual perspective image of FIG. 21 can reduce the stretching effectsin the below-horizon area. However, the cylindrical projection in theabove-horizon area of the image can cause the background objects 442 toappear to tilt toward the center of the image. In order to avoidintroducing this undesirable distortion effect, an imaging surface 464can be defined, where the imaging surface 464 is in the shape of ahorizontal quarter-cylinder for the below-horizon area of the image andis a flat planar shape for the above-horizon area of the image. A cameramodel including the imaging surface 464 has been shown to producesuperior virtual perspective view image synthesis.

Camera models with imaging surfaces other than those shown on FIG. 22can also be defined. For example, instead of a quarter-cylinder and aflat plane as in the surface 464, an image surface with a progressivelychanging radius of curvature could be defined. Also, image surfacecurvature effects from left to right could be introduced, along with thebottom to top curvature effects. Using various combinations such asthese, compound image surfaces can be defined which produce bothrealistic and visually pleasing virtual perspective views.

The second problem with a merged perspective view constructed fromleft-front and right-front camera images is a “double vision” effect inan overlapped field of view region, as seen in the double appearance oftree 448 in FIG. 21. The overlap area being described is shownapproximately as the overlap area 110 of FIG. 1. Blind spots—or areasnot covered by either the left or right camera image—can also exist,resulting in missing objects or texture in the merged view. Otherprocessing techniques, discussed below, can be applied to the virtualperspective view in order to correct discrepancies in the overlap areaor blind spot between images from the camera 102 and the camera 104.

One technique which can be applied to the overlap area betweenleft-front and right-front images is known as video morphing. In thevideo morphing technique, feature points are detected in the centeroverlap area of the images, and key points are registered. The left andright images are then morphed, meaning stretched or transformed, tocause the feature points to match up in the merged image. Previous imageframes and vehicle motion information can be used to enhance imagesynthesis in the overlap area, where some objects move from anon-overlap area of the image to the overlap area. Using both the imagemorphing and the previous image frames, a temporal video morphing can beapplied to the virtual perspective view which corrects bothdouble-imaging and blank spots in the overlap area.

Another technique which can be applied to the overlap area betweenleft-front and right-front images is image rendering based on 3D sceneestimation. This technique is similar to the method described above inthe flowchart diagram 380 of FIG. 18. As discussed above, this techniqueinvolves using stereo vision, structure from motion (SFM) and data fromother onboard sensors to construct a 3D model of the scene in front ofthe vehicle 100. The 3D object scene can be correlated to the objects inthe image, and the perspective view image can be re-rendered toeliminate the duplicate appearance of any objects, such as the tree 448.

FIG. 23 is an illustration of a perspective view in front of the vehicle100 in which the distortions seen in FIG. 21 have been corrected byapplication of the new virtual image surface model of FIG. 22, and theother techniques discussed above. It can be seen in FIG. 23 that thestretching effect in the area 444 is eliminated due to the applicationof the cylindrical camera image model in the below-horizon area. Thus,there is no loss of resolution in the area 444. It can also be seen inFIG. 23 that the vertical joint 446 appears vertical as it should, alsodue to the application of the cylindrical camera image model in thebelow-horizon area. Finally, the double image of the tree 448 iseliminated due to application of the overlap region correctiontechniques discussed above.

FIG. 24 is a flowchart diagram 480 of a method for creating an enhancedperspective view constructed from vehicle frontal camera images usingthe camera image surface model of FIG. 22 and other correctiontechniques. At box 482, a de-warping calculation is performed on thevirtual perspective view constructed from left- and right-front cameraimages, to account for the distortion when the naturally warped imagesfrom the wide-angle lens of the cameras 102 and 104 is flattened to makestraight lines appear straight. At box 484, the camera model imagesurface 464 is applied to the de-warped image, to correct stretching inthe below-horizon part of the image while preserving the verticalappearance of vertical objects in the above-horizon part of the image.At box 486, the area where the left and right camera images overlaps iscorrected to eliminate double-imaging of objects and blank spots. Thecorrections in the overlap area can include both video morphing and 3Dscene estimation techniques, as discussed above. At box 488, a finalvirtual perspective view is provided, including the correctionsdiscussed above, and as shown in FIG. 23.

Using the techniques described above, problems which are typicallyassociated with virtual perspective views synthesized from wide-anglecamera images using traditional camera models can be solved, resultingin a very realistic virtual front perspective view. The virtualperspective view synthesis techniques discussed above in the context ofa front view are also applicable to virtual perspective views to therear and to the sides of the vehicle 100, if camera images areavailable. The techniques can also be used with more than two camerasper virtual view. If more than two cameras are available to construct avirtual perspective front view, for example, the resultant view qualitywill improve because more image pixels will be used as input and imagede-warping effects can be reduced.

Enhanced front curb viewing systems provide drivers with visualinformation which would otherwise be unavailable. Front curb viewsprovide the driver with the assistance needed to accurately park theirvehicle in proximity to curbs and other frontal obstacles, and the peaceof mind of knowing that they are not going to accidentally bump into theobstacle and damage the vehicle. The automatic generation and display ofthese front curb views deliver value to the driver, resulting inincreased customer satisfaction with the vehicle, and potentiallyavoiding expensive repair bills.

The foregoing discussion discloses and describes merely exemplaryembodiments of the present invention. One skilled in the art willreadily recognize from such discussion and from the accompanyingdrawings and claims that various changes, modifications and variationscan be made therein without departing from the spirit and scope of theinvention as defined in the following claims.

What is claimed is:
 1. A system for providing enhanced perspectiveviewing of an area in front of a vehicle, said system comprising: afirst camera mounted at the left front of the vehicle; a second cameramounted at the right front of the vehicle; an image acquisition modulein communication with the first and second camera, said imageacquisition module receiving raw images from the cameras, and the rawimages from the cameras have a generally forward view perspective; aprocessor including a memory module, said processor being configured toprovide a virtual perspective view of the area in front of the vehicle,where the virtual perspective view is synthesized from the raw imagesfrom the cameras, the virtual perspective view uses a camera imagesurface model which corrects for artificial magnification and stretchingeffects of the image, and the virtual perspective view includescorrections in an overlap region of the raw images from the first cameraand the second camera; and a display unit in a cockpit area of thevehicle for displaying the virtual perspective view from the processorfor viewing by a driver of the vehicle.
 2. The system of claim 1 whereinthe camera image surface model is a model of a surface onto which pixelsfrom the raw images are projected, and the surface is in a shape of ahorizontal quarter-cylinder for a below-horizon area of the raw imagesand is a flat planar shape for an above-horizon area of the raw images.3. The system of claim 2 wherein the camera image surface model isconfigured such that vertical objects in front of the vehicle appear asvertical in the virtual perspective view.
 4. The system of claim 1wherein the processor uses temporal filling techniques to correct theoverlap region of the raw images, where the temporal filling techniquesuse actual image data from previous time samples, along with vehiclemotion data, to produce virtual image data for a current time in theoverlap region of the raw images.
 5. The system of claim 1 wherein theprocessor uses video morphing techniques to correct the overlap regionof the raw images, where the video morphing techniques identify featurepoints in the overlap region of the raw images and transform the rawimages so that the feature points co-align in the virtual perspectiveview.
 6. The system of claim 1 wherein the processor usesstructure-from-motion techniques to correct the overlap region of theraw images, where the structure-from-motion techniques use a sequence ofthe raw images and object data from a vehicle radar or lidar system tobuild three-dimensional models of objects in front of the vehicle, andthe three-dimensional models are used to eliminate double-imaging andblank spots in the overlap region of the raw images.
 7. The system ofclaim 1 wherein the virtual perspective view is displayed in a firstwindow on the display unit and a bird's-eye view is displayed in asecond window on the display unit, where the first window is larger thanthe second window, and the bird's-eye view includes a depiction of thevehicle in the center with a forward-view image in front of the vehicledepiction and a rear-view image behind the vehicle depiction.
 8. Thesystem of claim 1 wherein the virtual perspective view includes anindication of whether a curb in front of the vehicle will make contactwith a low-hanging body component of the vehicle.
 9. A method forproviding enhanced perspective viewing of an area in front of a vehicle,said method comprising: providing images from front-mounted cameras onthe vehicle, where the images have a generally forward view perspective;synthesizing a virtual perspective view from the images from thecameras; performing, using a microprocessor, a de-warping calculation onthe virtual perspective view to produce a de-warped virtual perspectiveview; enhancing the de-warped virtual perspective view by applying avirtual camera image surface model which corrects for artificialmagnification and stretching effects of image de-warping, creating anenhanced virtual perspective view; correcting, in the enhanced virtualperspective view, double-image and blank region discrepancies whichexist in an overlap area of the images from the front-mounted cameras;and displaying the enhanced virtual perspective view on a display devicefor viewing by a driver.
 10. The method of claim 9 wherein enhancing thede-warped virtual perspective view includes using the virtual cameraimage surface model which is a model of a surface onto which pixels fromthe images are projected, and the surface is in a shape of a horizontalquarter-cylinder for a below-horizon area of the images and is a flatplanar shape for an above-horizon area of the images.
 11. The method ofclaim 10 wherein the virtual camera image surface model is configuredsuch that vertical objects in front of the vehicle appear as vertical inthe enhanced virtual perspective view.
 12. The method of claim 9 whereincorrecting double-image and blank region discrepancies includes usingtemporal filling techniques to correct the overlap area of the images,where the temporal filling techniques use actual image data fromprevious time samples, along with vehicle motion data, to producevirtual image data for a current time in the overlap area.
 13. Themethod of claim 9 wherein correcting double-image and blank regiondiscrepancies includes using video morphing techniques to correct theoverlap area of the images, where the video morphing techniques identifyfeature points in the overlap area and transform the raw images so thatthe feature points co-align in the enhanced virtual perspective view.14. The method of claim 9 wherein correcting double-image and blankregion discrepancies includes using structure-from-motion techniques tocorrect the overlap area of the images, where the structure-from-motiontechniques use a sequence of the images and object data from a vehicleradar or lidar system to build three-dimensional models of objects infront of the vehicle, and the three-dimensional models are used toeliminate double-imaging and blank spots in the enhanced virtualperspective view.
 15. The method of claim 9 wherein the enhanced virtualperspective view is displayed in a first window on the display deviceand a bird's-eye view is displayed in a second window on the displaydevice, where the first window is larger than the second window, and thebird's-eye view includes a depiction of the vehicle in the center with aforward-view image in front of the vehicle depiction and a rear-viewimage behind the vehicle depiction.
 16. The method of claim 9 whereinthe enhanced virtual perspective view includes an indication of whethera curb in front of the vehicle will make contact with a low-hanging bodycomponent of the vehicle.
 17. A method for providing enhancedperspective viewing of an area in front of a vehicle, said methodcomprising: providing images from front-mounted cameras on the vehicle,where the images have a generally forward view perspective; synthesizinga virtual perspective view from the images from the cameras; performing,using a microprocessor, a de-warping calculation on the virtualperspective view to produce a de-warped virtual perspective view;enhancing the de-warped virtual perspective view by applying a virtualcamera image surface model which corrects for artificial magnificationand stretching effects of image de-warping, where the virtual cameraimage surface model has a quarter-cylindrical shape applied to abelow-horizon area and a flat planar shape applied to an above-horizonarea, creating an enhanced virtual perspective view; correcting, in theenhanced virtual perspective view, double-image and blank regiondiscrepancies in an overlap area of the images from the front-mountedcameras, including image rendering using video morphing andthree-dimensional scene estimation techniques; and displaying theenhanced virtual perspective view on a display device for viewing by adriver.
 18. The method of claim 17 wherein enhancing the de-warpedvirtual perspective view by applying a virtual camera image surfacemodel creates an enhanced virtual perspective view in which verticalobjects in front of the vehicle appear as vertical.
 19. The method ofclaim 17 wherein video morphing includes identifying feature points inthe overlap area and transforming the raw images so that the featurepoints co-align in the enhanced virtual perspective view, andthree-dimensional scene estimation includes using a sequence of theimages to build three-dimensional models of objects in front of thevehicle, and the three-dimensional models are used to eliminatedouble-imaging and blank spots in the enhanced virtual perspective view20. The method of claim 17 further comprising using temporal fillingtechniques to correct double-image and blank region discrepancies in theoverlap area of the images, where the temporal filling techniques useactual image data from previous time samples, along with vehicle motiondata, to produce virtual image data for a current time in the overlaparea.